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MEMORANDUM  FOR  PROGRAM  EXECUTIVE  OFFICER  FOR  INTELLIGENCE 

ELECTRONIC  WARFARE  AND  SENSORS 


SUBJECT:  Report  on  the  Development  Testing  of  Prophet  Mission-Critical  Software 
(Report  No.  D-2003-051) 


We  are  providing  this  report  for  your  review  and  comment.  We  considered 
management  comments  on  a  draft  of  this  report  when  preparing  the  final  report. 

The  Army  Program  Executive  Officer  for  Intelligence,  Electronic  Warfare  and 
Sensors  comments  were  responsive  to  the  intent  of  the  recommendations.  However,  we 
request  that  the  Program  Executive  Officer  provide  additional  information  showing  ’ 
completion  of  Recommendations  A.  and  B.  by  March  21,  2003. 

We  appreciate  the  courtesies  extended  to  the  evaluation  staff.  Questions  on 
the  evaluation  should  be  directed  to  Mr.  Kenneth  H.  Stavenjord  at  (703)  604-8952 
(DSN  664-8952)  or  Mr.  Peter  C.  Johnson  at  (703)  604-9601  (DSN  664-9601).  See 
Appendix  C  for  the  report  distribution.  The  team  members  are  listed  inside  the  back 
cover. 


David  K.  Steensma 
Deputy  Assistant  Inspector  General 
for  Auditing 


Office  of  the  Inspector  General  of  the  Department  of  Defense 


Report  No.  D-2003-XXX 

(Project  No.  D2001PT-0104) 


January  XX,  2003 


Evaluation  of  Development  Testing  of  Prophet 
Mission-Critical  Software 


Executive  Summary 


Who  Should  Read  This  Report  and  Why?  This  report  concerns  developmental 
testing  of  DoD  acquisition  programs,  which  should  be  of  particular  interest  to  program 
managers  and  acquisition  professionals. 

Background.  This  report  is  a  review  of  the  development  testing  of  mission-critical 
software  for  the  U.S.  Army  Prophet  Engineering  and  Manufacturing  Development 
(EMD)  System  and  the  Prophet  Block  I  System.  Prophet  is  a  Division-Level  ground 
based  electronic  surveillance  system,  which  provides  protection  in  a  direct  support  role 
to  the  maneuver  brigade;  either  stationary,  or  while  on  the  move.  The  system  monitors 
and  exploits  signals  of  interest  and  determines  the  area  of  signal  origin  by  providing 
direction  finding  and  line-of-bearing  in  the  frequency  range  of  20  Megahertz  to  2000 
Megahertz.  Prophet  is  operated  in  either  the  dismounted  or  mounted  mode.  In  the 
dismounted  mode  Prophet  is  man-packed  with  a  portable  antenna.  In  the  mounted 
mode  Prophet  is  installed  in  an  equipment  enclosure  carried  on  a  High  Mobility  Multi- 
Purpose  Wheeled  Vehicle  with  the  antennas  attached  to  a  retractable  mast. 

Prophet  EMD  was  developed  to  validate  the  operational  performance  of  the  system 
prior  to  a  decision  for  full  rate  production  and  deployment.  Prophet  EMD 
developmental  testing  was  completed  in  September  2000  and  in  general  met  the  test 
criteria.  Prophet  Block  I  is  the  full  rate  production  system  and  has  identical  electronics 
to  Prophet  EMD  but  also  includes  the  Man  Machine  Interface.  Prophet  Block  I  First 
Article  Performance  testing  was  from  February  2002  to  May  2002  and  as  a  result  some 
errors  were  found  and  are  being  corrected.  After  First  Article  Performance  Testing, 
three  Prophet  Block  I  systems  are  to  be  refurbished  and  sent  to  Titan  System 
Corporation,  Melbourne,  Florida,  for  final  sell  off  and  acceptance.  The  first 
production  contract  was  awarded  to  Titan  Systems  Corporation  in  June  2001 .  The 
contract  obligates  $7.7  million  for  non-recurring  engineering  work  and  six  pre- 
production  systems.  Additionally,  the  program  office  plans  to  procure  an  additional  83 
systems  at  a  recurring  cost  of  approximately  $300,000  per  system. 

Results.  Prophet  Block  I  with  the  Man  Machine  Interface  will  be  fielded  without 
ensuring  that  the  system  meets  operational  needs  and  that  it  retains  its  effectiveness  and 
suitability  for  the  typical  user  in  an  operational  environment.  For  details  of  this 
evaluation  result  see  finding  A  of  this  report. 


Independent  security  certification  analysis,  testing,  and  evaluation  are  not  planned. 
Prophet  will  be  fielded  without  knowing  the  extent  to  which  the  systems  meet 
information  assurance  requirements.  For  details  of  this  evaluation  result  see  finding  B 
of  this  report. 

Management  Comments.  The  Army  Program  Executive  Officer  for  Intelligence, 
Electronic  Warfare  and  Sensors  (PEO  IEWS)  partially  concurred  with  finding  A  and 
finding  B.  In  finding  A,  PEO  IEWS  agreed  that  an  operational  test  assessment  was 
required  for  Prophet  Block  I  with  Man  Machine  Interface  (MMI)  and  that  it  was 
scheduled  for  October  2002  but  disagreed  that  it  was  not  planned.  In  finding  B,  PEO 
IEWS  agreed  that  both  Prophet  Engineering  Manufacturing  and  Development  (EMD) 
and  Prophet  Block  I  should  be  designated  at  certification  Level  2  but  differed  on  the  IG 
DoD  selection  of  the  system  characteristic  alternatives  used  in  the  System  Security 
Authorization  Agreement,  and  disagreed  that  Prophet  Block  I  certification  level  had  not 
been  designated.  Although  it  was  not  required  to  comment,  PEO  IEWS  strongly  non- 
concurred  that  a  management  control  weakness  existed  relating  to  the  planning  of  the 
operational  test  assessment  for  Prophet  Block  I  with  MMI. 

Evaluation  Response.  Although  Army  PEO  IEWS  only  partially  concurred  with 
finding  A  and  finding  B,  the  comments  were  responsive.  PEO  IEWS  agreed  with  the 
recommendation  that  an  operational  test  assessment  of  Prophet  Block  I  with  MMI  was 
required  and  they  had  scheduled  the  test  for  October  2002.  PEO  IEWS  also  agreed 
with  the  recommendation  that  both  Prophet  EMD  and  Prophet  Block  I  should  be 
redesignated  at  certification  Level  2.  Both  systems  are  required  to  undergo  information 
assurance  analysis  and  testing  commensurate  with  Level  2  requirements.  Even  though, 
it  was  not  required  to  comment,  PEO  IEWS  strongly  non-concurred  with  the  comment 
regarding  the  adequacy  of  management  control  in  Appendix  A  related  to  finding  A. 
However,  we  determined  that  a  management  control  weakness  was  justified  based  on 
DoD  Instruction  5010.40  “Management  Control  Program  Procedures,”  August  28, 
1996.  Specifically,  the  weakness  found  was  that  an  operational  test  assessment  for 
Prophet  Block  I  with  MMI  was  not  adequately  planned  and  the  relative  impact  of  the 
weakness  on  testing,  deployment,  and  use  of  the  system  required  the  attention  of  the 
next  higher  level  of  management.  We  request  that  PEO  IEWS  provide  the  test  report 
documenting  the  results  of  the  operational  test  assessment  for  Prophet  Block  I  with 
MMI.  We  also  request  PEO  IEWS  provide  a  copy  of  an  updated  System  Security 
Authorization  Agreement  for  Prophet  Block  I  documenting  the  certification  level  of 
Prophet  Block  I  and  Prophet  EMD  as  well  as  the  security  certification  analysis  and  tests 
performed.  Both  documents  should  be  provided  no  later  than  February  XX,  2003. 
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Background 

Army  Regulation  73-1,  “Test  and  Evaluation  Policy,”  February  27,  1995,  and  DoD 
Regulation  5000. 2-R,  “Mandatory  Procedures  for  Major  Defense  Acquisition 
Programs  (MDAPS)  and  Major  Automated  Information  System  (MAIS)  Acquisition 
Programs,”  April  5,  2002,  govern  the  Prophet  program  development  testing  policy. 
Army  Regulation  73-1  describes  development  testing  such  as  engineering-type  tests 
used  to  verify  that  design  risks  are  minimized,  substantiate  technical  performance, 
and  certify  system  readiness  for  operational  test  and  evaluation.  DoD  5000. 2-R, 
states  that  development  test  and  evaluation  shall:  1)  Identify  the  technological 
capabilities  and  limitations;  2)  Identify  and  describe  design  technical  risks;  3)  Stress 
the  system  under  test;  4)  Address  the  potential  of  satisfying  Operational  Test 
and  Evaluation  requirements;  5)  Analyze  the  capabilities  and  limitations  of 
alternatives;  6)  Assess  progress  toward  meeting  Key  Performance  Parameters  and 
other  Operational  Requirements  Document  requirements;  7)  Assess  technical 
progress  and  maturity  against  critical  technical  parameters;  8)  Provide  data  and 
analytic  support  to  the  decision  process;  9)  In  the  case  of  Information  Technology 
systems,  support  the  information  systems  security  certification  process;  and,  10) 
Prior  to  full  rate  production,  demonstrate  the  maturity  of  the  production  process. 

Prophet  is  a  Division-Level  ground  based  electronic  surveillance  system,  which 
provides  protection  in  a  direct  support  role  to  the  maneuver  brigade;  either 
stationary,  or  while  on  the  move.  The  system  monitors  and  exploits  signals  of 
interest  and  determines  the  area  of  signal  origin  by  providing  direction  finding  and 
line-of-bearing.  Prophet  is  designed  to  detect  line-of-sight  radio  signals  in  the 
frequency  range  of  20  Megahertz  to  2000  Megahertz,  which  is  in  the  High 
Frequency,  Very  High  Frequency  and  Ultra  High  Frequency  bands.  Prophet  is 
operated  in  either  the  dismounted  or  mounted  mode.  In  the  dismounted  mode 
Prophet  is  man-packed  with  a  portable  antenna.  In  the  mounted  mode  Prophet  is 
installed  in  an  equipment  enclosure  carried  on  a  High  Mobility  Multi-Purpose 
Wheeled  Vehicle  with  the  antennas  attached  to  a  retractable  mast. 

The  program  office  has  performed  testing  on  two  systems,  Prophet  Engineering  and 
Manufacturing  Development  (EMD)  and  Prophet  Block  I.  Prophet  EMD  and 
Prophet  Block  I  have  identical  electronics  with  the  exception  that  Prophet  EMD  does 
not  include  a  subsystem  called  the  Man  Machine  Interface  (MMI).  Prophet  EMD 
was  developed  and  tested  to  validate  system  performance.  The  results  from  those 
tests  were  used  in  the  decision  to  proceed  with  full  rate  production  and  deployment 
of  Prophet  Block  I.  During  the  evaluation  we  selected  three  subsystems  in  Prophet 
Block  I  that  contain  mission-critical  software.  They  are  the  MD-405A 
Receiver/Processor,  the  Processor  System  Interface  Unit  (PSIU),  and  the  MMI. 

During  direction  finding  operation  the  MD-405A  Receiver/Processor  records  and 
reports  a  line-of-bearing  for  each  received  signal  or  frequency  at  the  end  of  the 
selected  integration  period.  The  MD-405A  Receiver/Processor  is  operated  either  by 
the  front  panel  or  through  a  remote  host  connected  to  the  serial  interface  port.  The 
unit  has  the  capability  to  scan  up  to  400  normal  channels  and  20  priority  channels. 
With  the  Prophet  antennas,  the  unit  is  capable  of  performing  direction  finding  on 
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line-of-sight  radio  frequency  emitters  in  the  range  of  20  Megahertz  to  2000 
Megahertz.  The  MD-405A  Receiver/Processor  is  already  in  use  as  part  of  the 
Special-Operations  Command  AN/PRD-13(V)2,  a  man-packed  radio  frequency 
direction  finding  system.  All  software  in  the  MD-405A  Receiver/Processor  is 
controlled  by  Titan  Signal  Products  Division’s  configuration  management  process. 
The  MD-405A  Receiver/Processor  has  approximately  73,000  lines  of  Assembly 
source  code  of  which  450  additional  lines  were  added  for  the  Prophet  mission. 

PSIU  is  a  message  exchanger  and  process  monitor.  The  PSIU  receives,  transmits 
and  monitors  serial  data  streams  between  subsystems.  The  PSIU  has  approximately 
13,000  lines  of  Assembly  source  code  of  which  9,500  lines  were  reused  from 
software  in  the  MD-405A  Receiver/Processor. 

MMI  is  a  laptop  computer  remotely  interfaced  to  the  MD-405A  Receiver/Processor 
through  the  PSIU.  The  MMI  provides  a  central  control  center  for  system  and 
receiver,  a  signal  map  display,  a  signal  parameter  list  display,  and  tools  to  store,  sort 
and  create  signal  parameter  and  voice  traffic  files.  MMI  application  code  is  a 
combination  of  commercial  software,  and  custom  developed  software  written  in 
C+  + .  The  MMI  has  over  200,000  lines  of  C+  +  source  code  of  which  110,000 
lines  were  reused,  40,000  lines  were  modified  and  50,000  lines  were  developed. 

Software  development  testing  for  Prophet  EMD  was  done  at  the  unit  and  system 
level.  Unit  level  testing  focused  on  the  correct  execution  of  software  as  determined 
by  correct  data  flow  and  hardware  control  operations .  System  level  testing  was  done 
to  measure  the  technical  performance  of  Prophet  EMD  in  the  mounted  and 
dismounted  modes.  In  general,  software  development  testing  of  Prophet  EMD  at 
both  the  unit  level  and  system  level  met  the  test  criteria.  During  development 
testing,  two  software  errors  were  found.  One  error  resulted  in  a  change  of  software 
in  the  PSIU  and  the  other  error  required  software  changes  in  the  Tactical  Navigation 
System.  Both  errors  were  corrected  and  retested. 

Since  Prophet  EMD  and  Prophet  Block  I  are  almost  identical,  additional 
development  testing  on  the  full  rate  production  system  was  not  performed.  To 
validate  whether  Prophet  Block  I  meets  the  contractual  performance  requirements  a 
First  Article  Performance  test  is  being  conducted.  First  Article  Performance  testing 
includes  unit  and  system  level  testing.  Unit  level  testing  consists  of  tracing  the 
execution  of  the  MMI  controller,  application  program  interface  and  driver  level 
software;  reviewing  the  data  messages  being  passed  between  the  MMI,  the 
MD-405A  Receiver/Processor  and  the  PSIU;  and  validating  the  operations  of  the 
MD-405A  Receiver/Processor  and  MMI.  After  completion  of  unit  level  testing, 
Prophet  Block  I  will  undergo  system  level  technical  and  environmental  verification 
tests.  According  to  the  program  office,  Prophet  Block  I  First  Article  Performance 
testing  was  completed  in  May  2002  and  as  a  result  some  errors  were  found  and  are 
being  corrected. 

The  first  production  contract  was  awarded  to  Titan  Systems  Corporation  in 
June  2001.  The  contract  obligates  $7.7  million  for  non-recurring  engineering  work 
and  six  pre-production  systems.  Additionally,  the  program  office  plans  to  procure 
an  additional  83  systems  at  a  recurring  cost  of  approximately  $300,000  per  system. 
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Objectives 

Our  objective  was  to  evaluate  development  testing  of  mission-critical  software  for 
Prophet  EMD  and  Prophet  Block  I.  Specifically,  we  evaluated  the  completeness 
and  adequacy  of  the  development  testing  of  mission-critical  software  in  the 
MD-405A  Receiver/Processor,  the  PSIU,  and  the  MMI  in  the  areas  of  planning, 
execution,  and  reporting.  We  also  evaluated  the  adequacy  of  responses  to  test 
results  of  Prophet  EMD  and  Prophet  Block  I,  as  well  as  how  testing  deficiencies 
affected  the  program. 
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A.  Operational  Test  Assessment  of 
Prophet  Block  I  with  the  Man 
Machine  Interface 

Prophet  Block  I  with  the  MMI  will  not  have  an  operational  test 
assessment  or  a  follow-on  operational  test  performed  prior  to  fielding. 
Initial  operational  testing  of  Prophet  EMD  did  not  include  the  MMI. 

The  program  office  has  not  planned  any  additional  operational  tests  prior 
to  fielding  because  key  criteria  were  met.  As  a  result,  Prophet  Block  I 
with  MMI  will  be  fielded  without  ensuring  that  the  system  meets 
operational  needs  and  that  it  retains  its  effectiveness  and  suitability  for 
the  typical  user  in  an  operational  environment. 

Requirements  for  Operational  Test  Assessment 

Prophet  acquisition  management  and  strategy  are  governed  by  Army  acquisition 
policy  contained  in  Army  Regulation  70-1,  “Army  Acquisition  Policy,” 
December  15,  1997.  Army  Regulation  70-1  follows  the  guidance  and 
procedures  contained  in  DoD  Directive  5000.1,  “The  Defense  Acquisition 
System,”  October  23,  2000,  and  DoD  Regulation  5000. 2-R,  “Mandatory 
Procedures  for  Major  Defense  Acquisition  Programs  (MDAPS)  and  Major 
Automated  Information  System  (MAIS)  Acquisition  Programs,”  April  5,  2002. 
DoD  5000. 2-R  requires  an  operational  test  assessment  when  there  are 
“hardware  and  software  alterations  that  materially  change  system  performance 
(operational  effectiveness  and  suitability),  which  includes  system  upgrades  and 
changes  to  correct  deficiencies  identified  during  test  and  evaluation.” 

Prophet  test  and  evaluation  is  also  governed  by  the  requirements  in  Army 
Regulation  73-1,  “Test  and  Evaluation  Policy,”  February  27,  1995.  Army 
Regulation  73-1  requires  follow-on  operational  tests  to  “refine  the  estimates 
made  during  initial  operational  testing,  provide  data  to  evaluate  changes,  verify 
that  deficiencies  in  materiel,  training,  or  concepts  have  been  corrected,  and 
provide  data  to  ensure  that  the  system  continues  to  meet  operational  needs  and 
that  it  retains  its  effectiveness  in  a  new  environment  or  against  a  new  threat.” 

Initial  Operational  Testing  of  Prophet  Engineering  and 
Manufacturing  Development  without  Man  Machine 
Interface 

An  operational  test  requires  testing  in  realistic  operational  environments  with 
users  who  represent  those  expected  to  operate  and  maintain  the  system  when  it 
is  fielded  or  deployed.  Initial  operational  testing  was  done  with  Prophet  EMD 
in  two  phases.  The  first  phase  was  used  to  measure  system  effectiveness, 
suitability  and  limited  survivability  when  employed  by  a  unit  in  an  operational 
matching  environment.  Phase  1  testing  also  included  an  assessment  used  to 
validate  the  mechanics  of  the  overall  information  processing  architecture  in  the 
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areas  of  tasking,  reporting  and  collection  management.  Phase  2  was  a  field  test 
exercise  in  conjunction  with  a  planned  brigade-level  exercise.  The  objective  of 
Phase  2  was  to  collect  data  on  operational  effectiveness  and  suitability.  Data 
collected  from  those  tests  determined  the  system  operational  effectiveness, 
suitability,  and  survivability  and  were  used  as  input  to  a  Milestone  III  full  rate 
production  and  fielding  decision.  But  those  tests  were  only  performed  on 
Prophet  EMD  and  did  not  include  the  MMI. 

Prophet  Block  I  First  Article  Performance  Testing  with  the 
Man  Machine  Interface 

Prophet  Block  I  First  Article  Performance  testing  with  the  MMI  is  being 
performed  at  the  contractor  facility  and  at  selected  subcontracted  laboratories 
and  test  ranges  with  the  participation  of  a  government  or  quality  assurance 
witness.  According  to  the  program  office,  First  Article  Performance  testing  was 
completed  in  May  2002  and  as  a  result  some  errors  were  found  and  are  being 
corrected.  During  those  tests,  Prophet  Block  I  had  system  performance  and 
environmental  verification  tests  using  manual  and  automated  methods.  System 
performance  testing  includes  demonstrating  MMI  mapping,  display  and  control 
functions,  signal  monitoring,  signal  detection  and  direction  finding  accuracy. 
Environmental  verification  testing  includes  demonstrating  reliability, 
maintainability  and  survivability.  Manual  testing  was  done  using  standard  test 
equipment.  Automated  testing  was  conducted  for  the  direction  finding  tests. 
After  First  Article  Performance  testing,  three  systems  are  to  be  refurbished  and 
sent  to  Titan  System  Corporation,  Melbourne,  Florida,  for  final  sell  off  and 
acceptance.  First  Article  Performance  testing  of  Prophet  Block  I  with  the  MMI 
did  not  include  an  operational  test  with  the  typical  user  in  an  operational 
environment. 

Operational  Testing  of  the  Man  Machine  Interface 

Prophet  Block  I  and  Prophet  EMD  system  electronics  are  the  same  with  the 
exception  that  Prophet  Block  I  also  includes  MMI.  The  MMI  provides  primary 
control  and  display  of  system  and  receiver.  Additionally,  MMI  provides  the 
operator  with  a  map  and  list  display  as  well  as  tools  to  analyze,  sort,  and  record 
data.  If  the  MMI  is  unavailable,  the  system  can  be  operated  using  manual 
controls.  Because  Prophet  EMD  initial  operational  tests  met  the  key 
performance  parameter  criteria,  the  program  office  decided  not  to  perform  any 
additional  operational  test  assessments  or  follow-on  operational  tests  prior  to 
fielding  of  Prophet  Block  I. 

The  MMI  is  a  laptop  computer  running  the  Microsoft  Windows  NT  4.0 
operating  system  and  contains  over  200,000  lines  of  source  code  developed  for 
electronic  surveillance  control.  The  electronic  surveillance  control  software 
consists  of  three  modules  that  are  used  for  control  interface,  application  program 
interface,  and  receiver  driver  interface.  The  MMI  also  contains  a  set  of 
commercial-off-the-shelf  software  modules  used  for  mapping,  data  tabling, 
digital  reporting,  databasing,  and  support.  In  comparison  with  the  other 
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subsystems  in  Prophet  Block  I,  MMI  has  the  largest  collection  of  commercial- 
off-the-shelf  and  custom  developed  software.  When  MMI  was  added  to 
Prophet,  the  PSIU  software  was  modified  to  distribute  data  to  and  from  MMI 
and  other  subsystems. 

Prophet  Block  I  with  MMI  does  improve  system  operational  performance  and 
effectiveness.  MMI  allows  easier  control  of  the  MD-405A  Receiver/Processor 
and  system,  has  better  displays  and  analysis  tools  for  examining  signals  of 
interest,  and  has  the  capability  to  store,  sort,  and  transfer  signal  data.  In  fact 
the  Prophet  system  manager  recognizes  the  operational  benefits  of  having  MMI 
because  planned  product  improvements  include  the  requirement  for  Prophet 
Block  III  to  have  MMI  for  intelligence  digital  reporting  and  databasing.  Even 
though  Prophet  can  perform  its  mission  without  MMI,  the  addition  of  MMI 
improves  system  performance  and  effectiveness.  If  an  operational  test 
assessment  or  a  follow-on  operational  test  is  not  performed  with  MMI,  possible 
operational  deficiencies  or  unsuitable  features  may  not  be  detected  and 
corrected. 

Summary 

The  Product  Manager,  Prophet  has  not  planned  for  Prophet  Block  I  with  MMI  to  have 
an  operational  test  assessment  or  a  follow-on  operational  test  performed  prior  to 
fielding.  Even  though  Prophet  can  perform  its  mission  without  MMI,  an  operational 
test  assessment  should  be  performed  because  with  MMI,  the  system  performance  and 
effectiveness  are  changed.  Specifically,  MMI  allows  easier  control  of  the  MD-405A 
Receiver/Processor  and  system;  has  better  displays  and  analysis  tools  for  examining 
signals  of  interest;  and  has  the  capability  to  store,  sort,  and  transfer  signal  parameters. 
As  a  result,  without  an  operational  test  assessment  or  follow-on  operational  test  of 
Prophet  Block  I,  no  operational  data  will  validate  the  performance,  effectiveness,  and 
suitability  of  MMI  for  the  typical  user  in  an  operational  environment. 

Recommendations,  Management  Comments  and  Evaluation 
Response 

A.  We  recommend  that  the  Product  Manager,  Prophet  plans  and  performs 
an  operational  test  assessment  of  Prophet  Block  I  by  updating  the  Test  and 
Evaluation  Master  Plan  and  executing  the  test  prior  to  fielding. 

Management  Comments.  The  Army  Program  Executive  Officer  for 
Intelligence,  Electronic  Warfare  and  Sensors  (PEO  IEWS)  partially  concurred 
with  the  finding.  PEO  IEWS  agreed  that  an  operational  test  assessment  of 
Prophet  Block  I  with  MMI  was  required  prior  to  fielding.  PEO  IEWS  disagreed 
that  an  operational  test  assessment  of  Prophet  Block  I  with  MMI  was  not 
planned  prior  to  fielding.  PEO  IEWS  stated  that  based  on  testing  to  date,  the 
Army  Test  and  Evaluation  Command  recommended  a  conditional  material 
release  for  fielding  of  Prophet  Block  I  with  MMI,  and  further  testing  was 
planned  prior  to  its  initial  fielding.  Specifically,  PEO  IEWS  stated  that  an 
operational  test  assessment  of  Prophet  Block  I  with  MMI  was  scheduled  for 
October  2002. 
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Evaluation  Response.  Although  PEO  IEWS  only  partially  concurred  with  the 
recommendations,  the  comments  were  responsive.  We  agree  with  the 
PEO  IEWS  statement  that  an  operational  test  assessment  of  Prophet  Block  I  with 
MMI  was  required.  However,  we  disagree  with  the  PEO  IEWS  comment  that 
an  operational  test  assessment  of  Prophet  Block  I  with  MMI  was  planned.  We 
concluded  this  because  an  operational  test  assessment  of  Prophet  Block  I  with 
MMI  was  not  documented  in  the  Prophet  Ground  Block  I  Test  and  Evaluation 
Master  Plan  Revision  5.0  or  the  Titan  Prophet  Production  Block  1  First  Article 
Performance  Test  Plan.  We  were  also  told  during  the  evaluation  that  for 
acceptance  and  first  article  tests  prior  to  fielding,  the  Army  Test  and  Evaluation 
Command  only  planned  to  participate  as  a  reviewer  and  commenter.  We 
request  that  PEO  IEWS  provide  the  test  report  documenting  the  results  of  the 
Prophet  Block  I  with  MMI  operational  testing. 
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B.  Certification  Level  2  for  Prophet 
Systems 

The  certification  of  the  Prophet  EMD  system  was  incorrectly  designated 
as  Level  1  instead  of  Level  2  because  the  alternatives  chosen  for  mission- 
reliance  and  integrity  did  not  match  the  system  characteristics.  In 
addition  the  Prophet  Block  I  production  system  certification  level  is  not 
yet  determined.  As  a  result,  independent  security  certification  analysis, 
testing,  and  evaluation  normally  conducted  for  Level  2  systems  are  not 
planned.  Prophet  will  be  fielded  without  knowing  the  extent  to  which  it 
will  meet  information  assurance  requirements. 

Information  Assurance  for  Prophet  Systems 

Information  Assurance.  Information  assurance  is  information  operations  that 
protect  and  defend  information  and  information  systems  by  ensuring  their 
availability,  integrity,  authentication,  confidentiality,  and  non-repudiation.  It 
includes  providing  for  the  restoration  of  information  systems  by  incorporating 
protection,  detection,  and  reaction  capabilities.  An  information  system  can  be 
any  computer  related  equipment  or  interconnected  system  or  subsystems  of 
equipment  that  is  used  in  the  acquisition,  storage,  manipulation,  management, 
movement,  control,  display  or  reception  of  voice  and  or  data  and  includes 
software,  firmware,  and  hardware.  The  information  systems  contained  in  the 
Prophet  systems  are  the  MD-405A  Receiver/Processor,  the  PSIU,  the  MMI,  and 
the  Tactical  Navigation  System.  In  order  to  ensure  that  the  Prophet  information 
systems  function  properly,  security  features  that  support  information  assurance 
must  be  implemented  and  tested. 

Prophet  System  Requirements  for  Availability,  Integrity,  Authentication, 
Confidentiality,  and  Non-Repudiation.  Availability  is  the  timely  and  reliable 
access  to  data  and  information  services  for  authorized  users.  All  Prophet 
information  systems  must  be  timely  and  reliable.  If  the  MD-405A 
Receiver/Processor  and  the  PSIU  are  untimely  or  unavailable  the  Prophet 
systems  will  not  be  able  to  meet  their  technical  performance  requirements. 

Also,  if  either  the  MMI,  or  the  Tactical  Navigation  System  is  untimely  or 
unavailable,  the  Prophet  Block  I  performance  and  effectiveness  will  be 
degraded. 

Integrity  is  the  condition  existing  when  data  is  unchanged  from  its  source  and 
has  not  been  accidentally  or  maliciously  modified,  altered,  or  destroyed. 

Without  data  integrity  the  Prophet  systems  would  not  be  able  to  accurately 
report  the  direction,  level,  and  frequency  of  signals  being  measured.  Therefore, 
all  Prophet  information  systems  must  ensure  that  the  integrity  of  data  used  or 
provided  is  unchanged. 

Authentication  is  a  security  measure  designed  to  establish  the  validity  of  a 
transmission,  message,  user,  or  system  or  a  means  of  verifying  an  individual’s 


8 


authorization  to  receive  specific  categories  of  information.  Prophet  Block  I  with 
MMI  must  have  authentication  measures  established  to  prevent  typical  users 
from  inadvertently  altering  data. 

Confidentiality  is  an  assurance  that  information  is  not  disclosed  to  unauthorized 
persons,  processes,  or  devices.  Non-repudiation  is  the  assurance  that  the  sender 
of  data  is  provided  with  proof  of  delivery  and  the  recipient  is  provided  with 
proof  of  origin,  so  neither  can  later  deny  having  processed  the  data.  Since  the 
Prophet  systems  are  stand-alone,  confidentiality  and  non-repudiation  are  not 
applicable. 

Requirements  for  System  Certification  Level 

DoD  5000.2-R,  “Mandatory  Procedures  For  Major  Defense  Acquisition 
Programs  (MDAPS)  and  Major  Automated  Information  Systems  (MAIS) 
Acquisition  Programs,”  April  5,  2002.  DoD  5000.2-R  Chapter  3.4.1  states 
that  development  test  and  evaluation  shall,  in  the  case  of  IT  systems,  support  the 
information  systems  security  certification  process,  and  DoD  5000.2-R  Chapter  3 
Section  6.1.3  states  that  information  assurance  testing  shall  be  conducted  on 
information  systems  to  ensure  that  planned  and  implemented  security  measures 
satisfy  Operational  Requirements  Document  and  System  Security  Authorization 
Agreement  requirements  when  the  system  is  installed  and  operated  in  its 
intended  environment. 

DoD  5200.40,  “Department  of  Defense  Information  Technology  Security 
Certification  and  Accreditation  Process  Instruction  (DITSCAP),”  December 
30,  1997.  DITSCAP  defines  a  process  that  standardizes  the  activities  leading  to 
system  security  accreditation,  and  applies  to  the  acquisition,  operation  and 
sustainment  of  any  DoD  system  that  collects,  stores,  transmits,  or  processes 
unclassified  or  classified  information.  Both  Prophet  EMD  and  Prophet  Block  I 
are  required  to  use  DITSCAP  since  both  contain  information  technology  systems 
that  collect  store,  transmit,  and  process  data.  The  DITSCAP  process  consists  of 
four  phases:  Phase  1,  Definition;  Phase  2,  Verification;  Phase  3,  Validation; 
and  Phase  4,  Post  Accreditation.  Information  collected  during  Phase  1  is  used 
to  determine  the  certification  level.  The  certification  level  establishes  the 
activities  that  are  performed  on  a  system  for  security  certification  and 
accreditation. 

DoD  8510. 1-M,  “Department  of  Defense  Information  Technology  Security 
Certification  and  Accreditation  Process  (DITSCAP)  Application  Manual,” 
July  31,  2000.  The  Application  Manual  supports  DITSCAP  by  presenting  a 
detailed  approach  to  the  activities  comprising  the  certification  and  accreditation 
process.  Chapter  3,  Phase  1  definition  provides  a  task  description  on  how  to 
determine  the  appropriate  certification  level  of  a  system.  Using  that  process  a 
certifier  determines  the  degree  of  confidentiality,  integrity,  availability,  and 
accountability  required  for  a  system  by  selecting  weighted  alternatives 
associated  with  each  system  characteristic.  The  weights  selected  are  then  totaled 
in  order  to  determine  the  appropriate  certification  level  of  the  system. 
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Certification  Level  2  Security  Test  and  Evaluation.  Depending  on  the  total 
score  derived  from  Phase  1 ,  the  possible  certification  levels  for  a  system  are 
Level  1,  which  requires  completion  of  the  minimal  security  checklist  that  either 
a  system  user  or  an  independent  certifier  may  complete;  Level  2,  which  requires 
the  completion  of  the  security  checklist  and  an  independent  certification 
analysis;  Level  3,  which  requires  the  completion  of  the  security  checklist  and  a 
more  in-depth  independent  analysis;  and  Level  4,  which  requires  the  completion 
of  the  security  checklist  and  the  most  extensive  independent  analysis.  If  a 
system  is  certified  at  Level  2,  a  system  security  test  and  evaluation  is  required 
during  Phase  3  validation.  The  purpose  of  the  security  test  and  evaluation  is  to 
validate  the  proper  integration  and  operation  of  all  security  features. 

Assessment  of  Prophet  Systems  Security  Certification  Level 

Prophet  Systems  Characteristic  Alternatives.  During  the  system  security 
definition  phase,  the  program  office  performed  a  certification  level  assessment 
of  Prophet  EMD.  Using  the  DITSCAP  Application  Manual  the  program  office 
selected  system  characteristic  alternatives  for  interface,  processing,  attribute, 
mission-reliance,  availability,  integrity,  and  information  categories.  The 
mission-reliance  alternative  chosen  was  “partial”  and  the  integrity  alternative 
chosen  was  “not  applicable.”  After  the  alternatives  were  selected  their 
corresponding  weights  were  added  and  compared  to  the  application  manual 
certification  level  table.  The  program  office  total  of  12  designated  Prophet 
EMD  at  certification  Level  1 .  The  program  office  did  not  assess  Prophet 
Block  I. 

During  our  evaluation  we  reviewed  the  characteristic  alternatives  selected  and 
determined  that  the  Prophet  EMD  and  Prophet  Block  I  alternative  for  mission- 
reliance  is  “total”  instead  of  “partial”  and  the  alternative  for  integrity  is  “exact” 
instead  of  “not  applicable.” 

Mission-Reliance.  Mission-reliance  relates  the  degree  to  which  the  success  of 
the  mission  relies  on  the  operation,  data,  infrastructure,  or  system.  The 
program  office  determined  that  Prophet  mission-reliance  is  “partial”  because  it 
concluded  that  the  mission  “can  be  accomplished  without  Prophet  Block  I  but 
with  greater  risk  to  personnel  equipment  and  mission  accomplishment.  ”  Our 
assessment  determined  that  Prophet  mission-reliance  is  “total”  because  the 
mission  of  the  system  is  totally  dependent  upon  the  specific  aspects  of  system, 
operation,  and  data.  In  particular,  Prophet  is  totally  dependent  on  the  operation 
and  data  of  the  MD-405A  Receiver/Processor  system,  Prophet  mounted  is 
totally  dependent  on  the  operation  and  data  of  the  PSIU  system,  and  when  MMI 
is  being  used  to  control  the  system  and  receiver  as  well  as  map  and  store  signals 
Prophet  Block  I  is  totally  dependent  on  the  operation  and  data  of  MMI.  With 
the  alternative  being  “total”  instead  of  “partial”  the  weight  for  the  mission- 
reliance  characteristic  is  seven  instead  of  three. 

Integrity.  Integrity  relates  the  degree  in  which  the  integrity  of  operation,  data, 
infrastructure,  or  system  is  needed  from  unauthorized  modification  or 
destruction  of  information.  The  program  office  determined  that  Prophet 
integrity  is  “not  applicable”  since  there  is  “no  Prophet  operation  that  would  risk 
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the  security  of  the  data  of  Prophet  and  that  no  malfunction  of  Prophet  would 
result  in  sending  data  out  of  the  system  that  from  a  security  standpoint,  should 
be  sent  out.”  We  determined  that  Prophet  integrity  is  “exact”  because  the 
specific  integrity  aspects  of  system,  operation,  and  data  must  be  exact  in  order 
for  the  system  to  accurately  report  the  direction,  level,  and  frequency  of  signals 
being  measured.  In  particular,  the  integrity  of  the  operation  and  data  of  the 
MD-405A  Receiver/Processor  must  be  exact  or  else  Prophet  would  not  be  able 
to  perform  its  mission.  Also  the  integrity  of  the  system,  operation,  and  data  as 
it  applies  to  PSIU  must  be  exact  or  else  Prophet  mounted  would  not  be  able  to 
perform  its  mission,  and  the  integrity  of  the  system,  operation,  and  data  as  it 
applies  to  MMI  must  be  exact  or  else  Prophet  Block  I  performance  will  be 
degraded.  With  the  alternative  being  “exact”  instead  of  “not  applicable,”  the 
weight  for  the  integrity  characteristic  is  six  instead  of  zero. 

The  following  table  depicts  the  evaluation  that  was  done  and  the  difference  in 
score  with  the  adjusted  values  in  mission-reliance  and  integrity. 


System  Characteristic  Comparison  Table  for  Prophet  Systems 


System 

SSAA 

Adjusted 

SSAA 

Adjusted 

Characteristic 

Alternative 

Alternative 

Weight 

Weight 

Interface  Mode 

Passive 

Passive 

2 

2 

Processing  Mode 

Dedicated 

Dedicated 

1 

1 

Attribute  Mode 

None 

None 

0 

0 

Mission-Reliance  * 

Partial 

Total 

3 

7 

Availability 

ASAP 

ASAP 

4 

4 

Integrity* 

Not-Applicable 

Exact 

0 

6 

Information 

Sensitive 

Sensitive 

2 

2 

Categories 

Total* 

12 

22 

Denotes  a  difference* 


Certification  Level.  The  total  score  determines  the  certification  level  of  the 
system.  A  score  of  less  then  16  designates  a  system  at  certification  Level  1.  A 
score  between  12  and  32  designates  a  system  at  certification  Level  2.  Because 
the  total  is  22  both  Prophet  EMD  and  Prophet  Block  I  should  be  at  certification 
Level  2. 

Because  both  Prophet  EMD  and  Prophet  Block  I  certification  levels  should  be  at 
Level  2,  the  Prophet  Program  Office  is  required  to  complete  a  minimal  security 
checklist,  perform  an  independent  certification  analysis  and  during  Phase  3 
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validation,  implement  a  security  test  and  evaluation.  By  implementing  those 
procedures,  system  security  features  for  availability,  integrity,  and 
authentication  would  be  verified  and  Prophet  would  operate  at  an  acceptable 
level  of  residual  risk. 

Summary 

Both  Prophet  EMD  and  Prophet  Block  I  information  systems  require 
information  assurance  security  features  to  ensure  system  availability,  integrity, 
and  authentication.  The  Prophet  Program  Office  performed  a  system  security 
assessment  of  Prophet  EMD  and  designated  it  at  certification  Level  1 .  The 
program  office  has  not  performed  an  assessment  of  Prophet  Block  I. 

Certification  Level  1  system  security  assessment  only  requires  a  minimal 
security  checklist.  During  our  assessment  we  determined  that  Prophet  EMD  and 
Prophet  Block  I  should  be  at  certification  Level  2.  Our  result  was  based  on 
Prophet  EMD  and  Prophet  Block  I  mission-reliance  being  “total”  instead  of 
“partial”  and  the  degree  of  integrity  being  “exact”  instead  of  “not  applicable.” 
Because  the  Prophet  systems  certification  is  not  Level  2,  an  independent  security 
certification  analysis,  testing,  and  evaluation  is  not  planned.  Prophet  will  be 
fielded  without  knowing  the  extent  to  which  the  systems  meet  information 
assurance  requirements. 

Recommendations,  Management  Comments  and  Evaluation 
Response 

B.  We  recommend  that  the  Product  Manager,  Prophet  designate  Prophet 
EMD  and  Prophet  Block  I  at  system  security  certification  Level  2  and 
implement  an  independent  security  certification  analysis  that  includes  a 
security  test  and  evaluation  during  Phase  3  validation  as  specified  in 
Department  of  Defense  Information  Technology  Security  Certification  and 
Accreditation  Process  (DITSCAP)  Application  Manual  8510. 1-M,  July  31, 
2000. 

Management  Comments.  The  Army  PEO  IEWS  partially  concurred  with  the 
finding.  PEO  IEWS  agreed  that  both  Prophet  EMD  and  Prophet  Block  I  should 
be  redesignated  at  certification  Level  2.  PEO  IEWS  also  agreed  that  both 
Prophet  EMD  and  Prophet  Block  I  are  required  to  have  information  assurance 
analysis  and  testing  commensurate  with  Level  2  requirements.  PEO  IEWS 
analysis  of  Prophet  EMD  system  characteristics  alternatives  differed  with  the 
system  characteristic  alternatives  selected  by  the  IG  DoD.  PEO  IEWS 
disagreed  that  the  Prophet  Block  I  certification  level  had  not  been  determined. 

Evaluation  Response.  Although  Army  PEIO  IEWS  only  partially  concurred, 
the  comments  were  responsive.  We  agree  that  both  Prophet  EMD  and  Prophet 
Block  I  should  be  redesignated  at  certification  Level  2.  We  also  agree  that  the 
appropriate  information  analysis  and  testing  should  be  performed  commensurate 
with  that  level.  The  differences  in  the  Army  PEO  IEWS  and  IG  DoD  selection 
of  system  characteristic  alternatives  is  not  an  issue  since  both  analysis  resulted 
in  the  determination  that  Prophet  EMD  is  designated  at  certification  Level  2. 

We  disagree  with  the  PEO  IEWS  statement  that  Prophet  Block  I  was  designated 
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at  certification  Level  2  during  the  evaluation  because  it  was  not  documented  in 
Phase  1  of  the  System  Security  Authorization  Agreement  for  Prophet  Block  I. 
We  request  that  PEO  IEWS  provide  the  updated  System  Security  Authorization 
Agreement  for  Prophet  Block  I  documenting  the  certification  level  of  Prophet 
Block  I  as  well  as  the  security  certification  analysis  and  tests  performed. 
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Appendix  A.  Scope  and  Methodology 


To  accomplish  the  evaluation  objective,  we  examined  program  management  of 
Prophet  Developmental  Testing  process  of  mission-critical  software  and  its 
related  documentation. 

We  reviewed  the  organizational  structure,  software  development  process  and 
software  development  testing  of  Prophet  EMD.  We  obtained  and  reviewed  the 
Prophet  Test  and  Evaluation  Master  Plan,  Developmental  Test  Plan,  and  Test 
Reports.  We  selected  three  of  the  six  subsystems  in  Prophet  that  contain 
mission-critical  software  for  our  evaluation.  We  visited  the  Prophet  Program 
Office,  the  developmental  and  operational  test  sites,  and  the  contractor  software 
developmental  test  facilities  to  verify  and  validate  test  process  and  test  results. 
Results  of  this  evaluation  provided  insight  into  the  completeness  and  adequacy 
of  the  developmental  software  testing  done  for  Prophet  EMD  and 
Prophet  Block  I. 

We  performed  this  evaluation  from  October  2001  to  May  2002  according  to 
standards  implemented  by  the  Inspector  General  of  the  Department  of  Defense. 
Our  scope  was  limited  in  that  we  did  not  include  tests  of  management  controls. 

We  visited  or  contacted  individuals  and  organizations  within  DoD,  Titan  System 
Corporation,  Santa  Clara,  California,  and  Tera  Research  Incorporated, 
Sunnyvale,  California. 

Use  of  Computer-Processed  data.  We  relied  on  computer-processed  data  from 
the  Command,  Control,  Communications,  Computers  and  Intelligence  Test 
Division  Final  Report  for  the  Production  Verification  Test  of  the  Prophet 
Block  I,  March  2000,  of  the  Army  Electronic  Proving  Ground  to  determine  the 
completeness  and  adequacy  of  the  testing.  We  reviewed  the  computer  hardware 
and  software  configurations  used.  We  observed  the  operations  of  the  test 
computer,  which  generated  the  test  signals,  as  well  as  examined  the  test  signal 
data  files.  Although,  we  did  not  perform  a  formal  reliability  assessment  of  the 
computer-processed  data,  nothing  came  to  our  attention  as  a  result  of  the 
procedures  that  caused  us  to  doubt  the  reliability  of  the  computer-processed 
data. 

General  Accounting  Office  High  Risk  Area.  The  General  Accounting  Office 
has  identified  several  high-risk  areas  in  the  DoD.  This  evaluation  report 
provides  coverage  of  the  Defense  Systems  Modernization  high-risk  area. 

Management  Control  Program  Review 

DoD  Directive  5010.38,  “Management  Control  (MC)  Program,”  August  26, 
1996,  and  DoD  Instruction  5010.40,  “Management  Control  (MC)  Program 
Procedures,”  August  28,  1996,  require  DoD  organizations  to  implement  a 
comprehensive  system  of  management  controls  that  provides  reasonable 
assurance  that  programs  are  operating  as  intended  and  to  evaluate  the  adequacy 
of  the  controls. 
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Scope  of  Review  of  Management  Controls.  Management  control  was  not  an 
announced  objective  of  this  evaluation.  However,  we  reviewed  the  management 
control  program  related  to  the  overall  evaluation  objectives  and  determined  that 
the  pertinent  management  controls  concerning  Developmental  and  Operational 
Testing  of  Mission-Critical  Software  for  Prophet  Block  I  system  were 
inadequate  (See  the  finding  A). 

Adequacy  of  Management  Control  Program.  We  identified  a  material 
management  control  weakness  at  Prophet  Program  Office  as  defined  by  DoD 
Instruction  5010.40.  The  Product  Manager,  Prophet  has  not  planned  for 
Prophet  Block  I  with  MMI  to  have  an  operational  test  assessment  or  a  follow-on 
operational  test  performed  prior  to  fielding.  Product  Manager,  Prophet  should 
plan  and  perform  an  operational  test  assessment  test  because  with  MMI,  the 
system  performance  and  effectiveness  are  changed.  If  management  implements 
all  recommendations,  the  management  control  weakness  will  be  corrected.  A 
copy  of  the  report  will  be  provided  to  the  senior  official  responsible  for 
management  controls  within  Program  Management  Office  for  Signal  Warfare. 

Adequacy  of  Management’s  Self-Evaluation.  Prophet  Program  Office  did  not 
identify  the  need  to  perform  the  operational  assessment  test  as  an  assessable  unit 
and,  therefore,  did  not  identify  or  report  the  material  management  control 
weaknesses  identified  by  the  audit. 

Management  Comments  on  Management  Control.  The  Army  PEO  IEWS 
strongly  non-concurred  with  the  IG  DoD  opinion  that  a  management  control 
weakness  existed  because  an  operational  test  assessment  was  not  planned  for 
Prophet  Block  I  with  MMI.  PEO  IEWS  stated  that  an  operational  test 
assessment  of  Prophet  Block  I  with  MMI  has  always  been  planned.  PEO  IEWS 
also  stated  that  management  controls  are  in  place  at  both  the  Program 
Management  and  Program  Executive  Office  levels,  which  include  program 
reviews,  planning,  and  an  acquisition  memorandum  that  requires  follow-on¬ 
testing  of  any  differences  from  the  Prophet  EMD  to  production  configuration. 

Evaluation  Response.  DoD  5010.40  “Management  Control  Program 
Procedures,”  August  28,  1996  states,  weaknesses  in  management  control  should 
be  reported  if  they  are  deemed  to  be  material.  A  material  weakness  in 
management  control  must  satisfy  two  conditions:  management  controls  are  not 
in  place,  or  are  not  used  or  are  inadequate;  and  the  weakness  identified  requires 
the  attention  of  the  next  higher  level  of  management.  During  the  evaluation  a 
material  weakness  in  management  control  related  to  finding  A  was  identified 
because  it  satisfied  the  two  conditions.  First  of  all,  management  controls  in 
place  did  not  adequately  plan  for  the  operational  test  assessment.  This  was 
determined  because  the  Prophet  Ground  Block  I  Test  and  Evaluation  Master 
Plan  Revision  5.0  and  the  Titan  Prophet  Production  Block  1  First  Article 
Performance  Test  Plan  did  not  document  the  planning  of  an  operational  test 
assessment  and  discussions  with  Army  Test  and  Evaluation  Command 
representatives  revealed  that  for  the  acceptance  and  first  article  tests  prior  to 
fielding  of  Prophet  Block  I,  the  Army  Test  and  Evaluation  Command  only 
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planned  to  participate  as  a  reviewer  and  commenter.  Secondly,  the  attention  of 
the  next  higher  level  of  management,  PEO  IEWS,  was  required.  This  was 
determined  because  of  the  relative  impact  of  the  material  weakness  on  the 
testing,  deployment,  and  use  of  Prophet  Block  I  with  MMI. 


Prior  Coverage 

No  prior  coverage  has  been  conducted  on  the  subject  for  the  last  5  years. 
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Appendix  B.  Definitions  of  Technical  Terms 


AN/PRD-13(V)2  -  Special  Operations  Command  radio  frequency  direction 
finding  system  consisting  of  the  MD-405A  Triple  Receiver/Processor,  direction 
finding  and  intercept  antennas,  cables  and  power  accessories. 

Assembly  -  A  programming  language  that  is  once  removed  from  a  computer’s 
machine  language. 

C  +  +  -  A  high-level  programming  language,  developed  by  Bjarne  Stroustrup  at 
Bell  Labs.  C+  +  adds  object-oriented  features  to  its  predecessor,  C. 

Line-Of-Bearing  -  A  line  extending  in  the  direction  of  a  bearing. 

Line-Of-Sight  -  A  straight  line  between  an  observer  and  a  target. 

MD-405A  -  An  integrated,  programmable,  intercept/direction  finding  processor 
that  supports  highly  automated  signal  intercept  and  direction  finding  functions 
over  the  frequency  range  from  100  kHz  to  2000  MHz. 

Megahertz  -  A  unit  of  frequency  equal  to  one  million  cycles  per  second. 

Prophet  Engineering  and  Manufacturing  Development  (EMD)  -  Prophet 
System  without  the  MMI  that  was  used  for  developmental  and  operational 
testing. 

Prophet  Block  I  -  Prophet  Full  Rate  Production  System  that  has  identical 
electronics  to  Prophet  EMD  with  the  addition  of  the  MMI. 

Root  mean  square  -  The  square  root  of  the  mean  value  of  the  sum  of  the 
squares. 

System  Level  Testing  -  Tests  used  to  assess  if  the  system  meets  the  overall 
performance  objectives  of  the  software  requirements  and  system  specifications. 

Unit  Level  Testing  -  The  lowest  level  developer  test  of  software.  The  purpose 
of  unit  testing  is  to  validate  requirements  expressed  in  the  detailed  design 
descriptions  and  software  requirements  specifications.  In  addition,  unit  testing 
is  performed  to  ensure  that  all  source  statements  in  a  unit  have  been  executed, 
each  conditional  branch  has  been  taken,  and  that  all  boundary  values  (for 
example,  minimum-maximum  values)  and  edit  criteria  are  tested. 
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Appendix  C.  Report  Distribution 


Office  of  the  Secretary  of  Defense 

Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics 
Under  Secretary  of  Defense  (Comptroller) 

Deputy  Chief  Financial  Officer 
Deputy  Comptroller  (Program/Budget) 

Deputy  Under  Secretary  of  Defense  (Command,  Control,  Communications  and 
Intelligence) 

Director,  Operational  Test  and  Evaluation 


Department  of  the  Army 

Assistant  Secretary  of  the  Army  (Financial  Management  and  Comptroller) 
Assistant  Secretary  of  the  Army  (Acquisitions,  Logistics,  and  Technology) 
Program  Executive  Officer  for  Intelligence,  Electronic  Warfare  and  Sensors 
Auditor  General,  Department  of  the  Army 


Department  of  the  Navy 

Naval  Inspector  General 

Auditor  General,  Department  of  the  Navy 


Department  of  the  Air  Force 

Assistant  Secretary  of  the  Air  Force  (Financial  Management  and  Comptroller) 
Auditor  General,  Department  of  the  Air  Force 


Non-Defense  Federal  Organization 

Office  of  Management  and  Budget 
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Congressional  Committees  and  Subcommittees,  Chairman  and 
Ranking  Minority  Member 

Senate  Committee  on  Appropriations 

Senate  Subcommittee  on  Defense,  Committee  on  Appropriations 
Senate  Committee  on  Armed  Services 
Senate  Committee  on  Governmental  Affairs 
House  Committee  on  Appropriations 

House  Subcommittee  on  Defense,  Committee  on  Appropriations 
House  Committee  on  Armed  Services 
House  Committee  on  Government  Reform 

House  Subcommittee  on  Government  Efficiency,  Financial  Management,  and 
Intergovernmental  Relations,  Committee  on  Government  Reform 
House  Subcommittee  on  National  Security,  Veterans  Affairs,  and  International 
Relations,  Committee  on  Government  Reform 
House  Subcommittee  on  Technology  and  Procurement  Policy,  Committee  on 
Government  Reform 
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Army  Program  Executive  Office  for  Intelligence, 
Electronic  Warfare  and  Sensors  Comments 


DEPARTMENT  OF  THE  ARMY 

PROGRAM  EXECUTIVE  OFFICE 
INTELLIGENCE.  ELECTRONIC  WARFARE  AND  SENSORS 
FORT  MONMOUTH,  NEW  JERSEY  07703-5301 

"£J»lY  TO 
ATTENTOW  Ol 

SFAE-lliW&S 

MEMORANDUM  FOR  United  States  Department  of  Defense,  Office  of  the  Inspector  General.  ATTN 
Mr.  David  A.  Brinkman,  Room  801, 400  Army  Navy  Drive.  Arlington.  VA  22202 

SUBJECT:  Response  to  Draft  DoD  IG  Report:  "Evaluation  of  Developmental  Testing  of  Prophet 
Mission-Essential  Software  (Project  No,  D200IPT-0104)” 


1 .  PEO  IEW&S  appreciates  the  opportunity  to  review  and  comment  on  the  draft  audit  report  on  Prophet 
As  requested,  enclosed  are  management  comments  on  the  DoD  IG  findings  and  recommendations. 

a.  1  partially  concur  with  A  -  Operational  Test  Assessment  of  Prophet  Block  I  with  the  Man  Machine 
Interface.  I  non-concur  with  the  finding  that  PM  Prophet  had  not  planned  for  Prophet  Blk  1  with  Man 
Machine  Interface  (MMI)  to  have  an  operational  test  assessment  of  the  prior  to  system  fielding,  Based  on 
testing  conducted  to  date,  the  Army  Test  Sc  Evaluation  Command  (ATF.C)  supports  a  conditional  materiel 
release  for  fielding  of  the  Prophet  Blk  1  with  MMI.  and  further  testing  is  planned  prior  to  its  initial 
fielding.  I  concur  with  the  recommendation  that  an  operational  test  assessment  is  required  prior  to 
fielding,  and  this  is  being  done. 

b.  1  partially  concur  with  B  -  Certification  Level  2  for  Prophet  Systems.  1  concur  with  the  finding  that 
the  Prophet  EMD  system  should  be  designated  as  certification  level  2,  but  for  different  reasons  than  the 
DoD  IG.  The  PM  has  rc-lookcd  the  Information  Assurance  certification  levels  and  had  input  changes  to 
the  SSAA.  I  non-concur  with  the  finding  that  the  Prophet  Block  I  certification  level  is  not  yet  determined, 
and  that  it  will  be  fielded  without  knowing  the  extent  to  w  hich  it  will  meet  information  assurance 
requirements.  Prophet  Blk  I  is  certification  level  2,  and  appropriate  security  certification  analysis  and 
accreditation  testing  has  been  performed.  1  concrn  with  the  recommendation  that  both  EMD  and 
production  configurations  should  be  designated  at  system  security  level  2,  and  that  IA  analysis  and  testing 
commensurate  with  that  level  be  performed. 

2  I  strongly  non  concur  with  the  audit  determination  that  identified  a  management  control  weakness 
bated  on  the  fioD  IG  finding  related  to  operational  lest  assessment  of  Prophet  Blk  I  with  MMI. 
Managcmem  controls  arc  in  place  both  at  the  PM  and  PEO  level.  These  include TIWG  participation  for 
test  planning,  frequent  program  reviews  with  die  PEO  that  include  test  program  status,  PM  monthly 
significant  activity  reports,  and  an  acquisition  decision  memorandum  that  requires  conduct  of  follow-on 
testing  of  any  differences  from  the  Prophet  EMD  to  production  configuration. 

3.  The  point  of  contact  for  this  matter  is  LTC  Bill  Stevenson,  Product  Manager,  Prophet,  at  commercial 
732-427-1479,  DSN  987-1479  or  email:  Bill.Stevenson@icws.monmouth.army.mil. 


Enel 

as  Program  Executive  Officer 

Intelligence,  Electronic  Warfare  and  Sensors 


PM  Prophet  Comments  on 

Draft  DoDIC;  Report:  Evaluation  of  Developmental  Testing  of 
Prophet  Mission-Essential  Software  (Project  No.  D2001PT-0104) 

Finding  A  Summary  :  Operational  Test  Assessment  of  Prophet  Block  I  with  the  Man 
Machine  Interface.  The  Product  Manager,  Prophet  has  not  planned  for  Prophet  Block  I  with 
MMK  to  have  an  operational  test  assessment  or  follow-on  operational  test  performed  prior  to 
fielding.  Even  though  Prophet  can  perform  its  mission  without  MM1,  an  operational  test 
assessment  should  be  performed  because  with  MM  I,  the  system  performance  and  effectiveness 
arc  changed.  Specifically,  MM  I  allows  easier  control  of  the  MD-405A  Rcceiver/Processor  and 
system;  has  better  displays  and  analytical  tools  for  examining  signals  of  interest;  and  has  the 
capability  to  store,  sort,  and  transfer  signal  parameters.  As  a  result,  without  an  operational  test 
assessment  or  follow-on  operational  test  of  Prophet  Block  I,  no  operational  data  will  validate  the 
performance,  effectiveness,  and  suitability  of  MMI  for  the  typical  user  in  an  operational 
environment. 

Recommendation:  We  recommend  that  Product  Manager,  Prophet  plan  and  perform  an 
operational  test  assessment  of  Prophet  Block  I  by  updating  the  Test  and  Evaluation  Master  Plan 
and  executing  the  test  prior  to  fielding. 

PMPfuphet  Position:  Partially  Concur.  Follow-On  Test  and  Evaluation  of  the  Prophet  Blk  I 
production  system  with  MMI  was  in  fact  planned  pnor  to  fielding.  An  operational  evaluation  of 
the  MMI  was  scheduled  for  8-12  July  02  with  the  Operational  Test  Command.  Howrevcr, 
soldiers  slated  to  support  the  operational  evaluation  were  unavailable  as  they  were  redirected  to 
the  Prophet  EMD  Quick  Reaction  Capability  (QRC)  tieldings  in  support  of  real-world  counter¬ 
terrorism  operations.  Due  to  this  reason  operational  testing  was  postponed.  An  operational 
assessment  of  the  MMI  is  planned  in  Oct-Nov  02  prior  to  the  first  scheduled  Prophet  Blk  I 
fielding.  DT  testing  of  the  production  Prophet  system  was  successfully  performed  from  1 7  June 
to  12  July  02.  Subsequent  to  this,  a  Prophet  Logistics/Maintenance  Demo  was  successfully 
conducted  and  US  Army  Test  and  Evaluation  Center  (ATEC)  logistics  and  sustainability 
evaluators  assessed  maintenance  BIT  and  diagnostic  functions  of  the  MMI.  The  ATEC  Material 
Release  memo  dated  20  September  02  states  that  “ATEC  supports  a  Conditional  Material 
Release  (C  MR)  for  Prophet  Block  I  (AN/MLQ-40(v)3,  which  is  configured  with  the  MD-405A 
receiver  set,  sensor  vchicle(s)  and  MMI  laptop  computer." 

The  current  plan  is  to  address  Prophet  Blk  I  operational  test  requirements  via  a  continuous 
system  evaluation  test  strategy  of  Prophet  system  upgrades.  This  strategy  has  been  coordinated 
within  the  Prophet  l  est  and  Integration  Working  Group  (TIWG).  The  first  part  of  the  test 
strategy  includes  an  operational  assessment  of  the  MMI  with  soldiers  at  Fort  Iluachuca 
(Electronic  Proving  Grounds)  using  Prophet  in  a  tactically  relevant  environment  and  is  currently 
scheduled  for  October  2002,  prior  to  initial  fielding  of  Prophet.  After  operation  of  the  system, 
soldiers  will  complete  a  survey  from  ihe  U.S.  Army  Test  and  Evaluation  Command’s  (ATEC) 
Prophet  system  Evaluator  and  MANPRINT  evaluator.  This  survey  asks  soldiers  to  compare  the 
effectiveness  and  suitability  of  the  AN  PRD- 1 3  (v)  2  front  panel  interface  to  the  new  laptop 
based  MMI  in  supporting  the  operator’s  performance  of  the  Prophet  SIGINT.  Next,  in  January 
2003  in  conjunction  with  fielding  to  the  First  Stryker  Brigade  Combat  Team  (SBCT)  at  Fort 
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Lewis,  operators  will  again  be  surveyed  at  the  conclusion  of  the  three-week  training  and  a  field 
exercise  period.  The  SBCT  soldiers  have  had  Prophet  EMD  systems  for  two  years  and  as  such 
are  highly  qualified  to  assess  the  effectiveness  and  suitability  of  the  MMI  upgrade  in  the  Prophet 
Production  system. 

The  Training  and  Doctrine  Command  System  Manager  Prophet  (TSM  Prophet)  and  the  Product 
Manager  Prophet  are  currently  updating  the  Test  and  Evaluation  Master  Plan  (TEMP).  This 
update  is  required  to  incorporate  the  ATEC  agreed  upon  Prophet  Continuous  System  Evaluation 
Strategy  and  changes  from  the  Annual  TSM  Prophet  ORD  update.  This  revised  test  strategy  will 
require  and  evaluate  the  MMI  at  least  three  times  in  the  next  eight  months  as  well  as  in  units  as 
the  systems  are  fielded  to  the  Army. 

PM  Prophet  is  supporting  ATEC’s  performance  of  an  operational  assessment  of  the  MMI  and 
feels  that  adequate  testing  of  the  MMI  has  and  is  being  performed  albeit  delayed  slightly  from 
that  which  was  originally  planned. 

Finding  Summary  B;  Certification  Level  2  for  Prophet  Systems.  Both  Prophet  EMD  and 
Prophet  Blk  I  information  systems  require  information  assurance  features  to  ensure  system 
availability,  integrity,  and  authentication.  The  Prophet  Program  Office  performed  a  system 
security  assessment  of  Prophet  EMD  and  designated  it  at  certification  Level  1.  The  program 
office  has  not  performed  an  assessment  of  Prophet  Blk  I.  Certification  Level  1  system  security 
assessment  only  requires  a  minimal  security  checklist.  During  our  assessment  we  determined 
that  Prophet  EMD  and  Prophet  Blk  I  are  at  certification  Level  2.  Our  result  was  based  on 
Prophet  EMD  and  Prophet  Blk  I  mission-reliance  being  “total”  instead  of  “partial”  and  the 
degree  of  integrity  being  “exact"  instead  of  “not  applicable.”  Because  the  Prophet  system’s 
certification  is  not  Level  2,  an  independent  security  certification  analysis,  testing,  and  evaluation 
is  not  planned.  Prophet  will  be  fielded  without  knowing  the  extent  to  which  the  systems  meet 
information  assurance  requirements. 

Recommendation:  We  recommend  that  Product  Manager,  Prophet  designate  Prophet  EMD  and 
Prophet  Blk  I  at  system  security  level  2  and  implement  an  independent  security  certification 
analysis  that  includes  a  security  test  and  evaluation  during  Phase  3  validation  as  specified  in 
Department  of  Defense  Information  Technology  Security  Certification  and  Accreditation  Process 
(DITSCAP)  Application  Manual  8510.1-M,  July  31, 2000. 

PM  Pronhet  Position:  Partially  Concur.  Prophet  will  be  fielded  with  sufficient  confidence 
that  the  system  meets  its  information  assurance  requirements.  The  Prophet  Blk  I SSAA  provided 
to  the  DoDIG  was  for  the  Prophet  EMD  system,  which  operates  at  the  sensitive  but  unclassified 
(SBU)  level.  This  original  certification  level  1  determination  aligned  with  the  operational 
classification  of  SOCOM’s  Improved  SOF  SIGINT  Manpack  System  (ISSMS)  also  known  as  the 
AN/PRD-1 3(v)2  which  is  the  SIGINT  sensor  component  of  the  current  Prophet  system.  The 
Prophet  EMD  configuration  (without  the  tape  recorder)  is  operating  today  at  SBCT  Ft.  Lewis 
and  at  QRC  sites  at  the  SBU  level.  PM  Prophet  has  since  recognized  that,  in  accordance  with 
NSA  classification  guidance  and  the  new  Prophet  Security  Classification  Guide  (SCG),  the 
system  must  be  accredited  at  the  collateral  SECRET  level  to  allow  for  inclusion  of  a  tape 
recorder  in  the  system.  The  Prophet  EMD  systems  are  currently  operating  under  an  “Interim 


23 


Approval  to  Operate”  (IATO),  and  will  be  accredited  as  part  of  the  final  Production  SSAA 
accreditation  in  IQ  FY03. 


The  Prophet  EMD  configuration  (without  the  tape  recorder)  was  originally  scored  a  12 
(certification  level  1).  The  entire  scoring  for  both  the  Prophet  EMD  and  Production  systems  was 
re-looked  when  the  SSAA  was  updated  for  Blk  I  Production.  The  level  1  certification  for  Prophet 
EMD  configuration  (without  the  tape  recorder)  did  not  change  and  remains  appropriate  with  a 
modified  score  of  1 5  (an  increase  of  3  based  on  Integrity).  To  allow  for  inclusion  of  a  tape 
recorder  in  either  the  EMD  or  production  configurations,  PM  Prophet  determined  the  score  to  be 
an  18,  which  places  the  system  at  certification  level  2.  The  DoDIG,  upon  review;  scored  the 
EMD  system  a  22,  also  certification  level  2,  which  is  consistent  with  fire  PM.  The  classification 
scoring  table  below  compares  the  updated  SSAA  scoring  against  the  DoDIG  recommendation. 


System  Characteristic 


SSAA  Scoring 


DoDIG  Recommendation 


Interface  Mode 
Processing  Mode 
Attribution  Mode 
Mission  Reliance  1 
Availability 
Integrity  2 

Information  Category ' 


Passive  -  2 
Dedicated  - 1 
None  -  0 
Partial  -  3 
ASAP -4 
Approximate  -  3 
Secret -5 
Total- 18 


Passive  -  2 
Dedicated  - 1 
None  -  0 
Total -7 
ASAP -4 
Exact  -  6 
Sensitive  -  2 
Total -22 


The  Prophet  Production  system  SSAA  documents  the  process  by  which  the  Production  system 
was  analyzed,  tested  and  evaluated  as  required  for  a  Level  2  system,  including  an  independent 
security  certification  analysis.  Security  accreditation  testing  was  performed  April  -  June  ’02. 

Finding  in  Appendix  A  Management  Control  Program  Review.  The  audit  identified  the 
lack  of  planning  on  the  PM’s  part  and  failure  to  identify  the  need  for  an  operational  test 
assessment  of  Prophet  Blk  I MMI  as  a  material  management  control  weakness. 

PM  Prophet  Position:  Non-Concur.  PM  Prophet  has  adequate  management  controls  and  had 
always  planned  for  operational  testing  of  the  MMI  prior  to  fielding.  As  discussed  above  in  the 
response  to  the  DoDIG’s  Finding  A,  PM  Prophet  has  been  continually  working  with  ATEC 
through  the  Prophet  Test  and  Integration  Working  Group  (TIWG)  to  plan,  define,  resource,  and 
execute  its  test  requirements.  PM  Prophet  feels  that  adequate  coordination  has  been  done  with 
ATEC  to  ensure  necessary  testing  is  performed. 


1  Mission  Reliance.  The  PM  rating  remains  unchanged.  The  original  PM  designation  of  mission  reliance  level  as 
Partial  remains  appropriate,  reflecting  that  the  mission  is  partially  dependent  on  the  system,  and  that  a  unit  could 
accomplish  tbeir  Force  Protection  mission  if  needed  without  Prophet 

2  Integrity.  The  PM  rating  has  been  revised.  The  Prophet  system  provides  approximate  measurement  of  the 
parametrics  including  DF  of  a  signal,  PM  Prophet  believes  that  “Approximate”,  rather  than  "Exact"  is  the  most 
appropriate  rating. 

5  Infonnation  Category.  The  PM  rating  has  been  revised.  In  order  to  incorporate  the  voice  tape  recorder  into  the 
system,  PM  Prophet  has  raised  the  operational  security  level  from  Sensitive  but  unclassified  (SBU)  to  Secret. 
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